DataProc
Example
Run on Google cloud's dataproc:
gcloud dataproc batches submit \
--project your-project \
--region us-central1 \
spark \
--version 1.2 \
--subnet default \
--class com.pany.spark.SomeSparkJob \
--jars gs://your-bucket/definity-spark-agent-X-X.jar \
--properties spark.plugins=ai.definity.spark.plugin.DefinitySparkPlugin,spark.definity.server=https://app.definity.run,spark.definity.api.token=$DEFINITY_API_TOKEN,spark.definity.env.name=demo,spark.definity.pipeline.name=example_pipeline
Compatibility matrix
| Dataproc Image | Spark Version | Scala Version | Definity Agent |
|---|---|---|---|
| 2.3 | 3.5.3 | 2.12.18 | 3.5_2.12-latest |
| 2.2 | 3.5.3 | 2.12.18 | 3.5_2.12-latest |
| 2.1 | 3.3.2 | 2.12.18 | 3.3_2.12-latest |
| 2.0 | 3.1.3 | 2.12.14 | 3.1_2.12-latest |
| 1.5 | 2.4.8 | 2.12.10 | 2.4_2.12-latest |
| 1.4 | 2.4.8 | 2.11.12 | 2.4_2.11-latest |
| 1.3 | 2.3.4 | 2.11.8 | 2.3_2.11-latest |